Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor S3 store creation #76

Merged

Conversation

simonbohnen
Copy link
Contributor

@simonbohnen simonbohnen commented Jan 19, 2023

  • Move URL parsing logic of S3FSStores (replacing Boto3Store and BotoStore) to the S3FSStore
  • Deprecated old store creation methods

If the changes in this PR look good, I will create separate PRs for the other stores from #71.

  • Added a docs/changes.rst entry

Copy link
Contributor

@janjagusch janjagusch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getting there!

Comment on lines 120 to 127
if old_wrappers:
warnings.warn(
"""
Using wrappers as part of the scheme is deprecated. Please specify wrappers as part of the fragment instead,
e.g. "s3://...#wrap:readonly+urlencode" instead of "s3+readonly+urlencode://...".
""",
DeprecationWarning,
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was already previously deprecated, right? It's nothing we've introduced in this PR, right?

Copy link
Contributor Author

@simonbohnen simonbohnen Jan 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was no warning given previously, so we should probably keep it like that for now. I removed the warning 👍🏼

minimalkv/_get_store.py Outdated Show resolved Hide resolved
minimalkv/net/s3fsstore.py Outdated Show resolved Hide resolved
tests/bucket_manager.py Show resolved Hide resolved
@simonbohnen simonbohnen requested review from janjagusch and removed request for xhochy, fjetter, siboehm and mlondschien January 26, 2023 13:55
environment.yml Outdated Show resolved Hide resolved
minimalkv/_get_store.py Show resolved Hide resolved
minimalkv/_get_store.py Outdated Show resolved Hide resolved
minimalkv/_get_store.py Outdated Show resolved Hide resolved
minimalkv/_get_store.py Outdated Show resolved Hide resolved
minimalkv/_get_store.py Outdated Show resolved Hide resolved
@simonbohnen simonbohnen requested a review from xhochy February 6, 2023 15:37
@xhochy xhochy merged commit e5ab466 into data-engineering-collective:main Feb 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants